HEADSS: HiErArchical Data Splitting and Stitching software for non-distributed clustering algorithms

The increase in data volume is challenging the suitability of non-distributed and non-scalable algorithms, despite advancements hardware. An example this challenge clustering. Considering that optimal clustering algorithms scale poorly with increased or are intrinsically non-distributed, accurate large datasets increasingly resource-heavy, relying on substantial expensive compute nodes. This scenario forces users to choose between accuracy scalability. In work, we introduce HiErArchical Data Splitting Stitching (HEADSS), a Python package designed facilitate at scale. By automating splitting stitching, it allows repeatable handling, removal, edge effects. We implement HEADSS conjunction HDBSCAN, where achieve orders magnitude reduction single node memory requirements for both distributed implementations, latter offering similar order reductions total run times while recovering analogous accuracy. Furthermore, our method establishes hierarchy features by using subset split data.1